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Abstract 

The phenomenon of replica symmetry breaking is investigated for the retrieval 
phases of Hopfield-type network models. The basic calculation is done for the gener- 
alized version of the standard model introduced by Horner [1] and by Perez- Vicente 
and Amit [2] which can exhibit low mean levels of neural activity. For a mean ac- 
tivity a = 1/2 the Hopfield model is recovered. In this case, surprisingly enough, we 
cannot confirm the well known one step replica symmetry breaking (1RSB) result 
for the storage capacity which was presented by Crisanti, Amit and Gutfreund [3] 
( a iRSB ^ 0.144). Rather, we find that 1RSB- and 2RSB-Ansatze yield only slightly 
increased capacities as compared to the replica symmetric value (qJ rsb ~ 0.138 186 
and a^ RSB ~ 0.138187 compared to a^ s ~ 0.137905), significantly smaller also 
than the value af m = 0.145 ± 0.009 reported from simulation studies. These values 
still lie within the recently discovered reentrant phase [4]. We conjecture that in 
the infinite Parisi-scheme the reentrant behaviour disappears as is the case in the 
SK-spin-glass model (Parisi-Toulouse-hypothesis). The same qualitative results are 
obtained in the low activity range. 



*supported by Sonderforschungsbereich 123 'Stochastische Mathematische Modelle' 
t supported by a Heisenberg fellowship 
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1 Introduction 

The Hopfield model of a neural network [5] is at present considered to be well understood. 
By suitably adopting mean field theory of spin glass like systems [6], Amit et al. [7] 
were able to compute its phase diagram in a replica symmetric (RS) approximation. In 
particular, they found the T = storage capacity of the model to be a RS ~ 0.138 in the RS 
framework. From simulations, they obtained a slightly larger value a s c im = 0.145 ± 0.009 
[7], and they conjectured that the origin of such a discrepancy might be put down to 
effects of replica symmetry breaking (RSB). 

A subsequent one-step replica symmetry breaking (1RSB) analysis of Crisanti et al. 
[3] did indeed yield aJ RSB ~ 0.144 in very good agreement with the earlier simulation 
results, so that the question of the storage capacity of this model appeared to have been 
settled: effects of RSB are found to be small and 1RSB corrections seem to bridge the 
gap between RS results and those of numerical simulations. Large scale simulations by 
Kohring [8] seemed to confirm the overall picture. 

For generalized Hopfield-type models adapted to store an ensemble of low-activity 
patterns [1,2], deviations between results of RS mean field analysis and simulation data 
were found to be much stronger. For instance, at an intermediate level a = 0.1 for the 
mean activity, considered to be in a neurophysiologically acceptable range, it was found 
that a RS ~ 0.483 [1], whereas simulations [9] yielded af n = 0.585±0.01, i.e. a discrepancy 
of roughly 20 % as opposed to a 4 % discrepancy in the standard model. 

The question then arises, whether in such a situation, a 1RSB analysis would likewise 
be sufficient to close the gap between RS results and estimates obtained from Monte-Carlo 
simulations, or whether - on contrary - further steps in Parisi's approximating scheme 
[10] or even the full hierarchical scheme of RSB [11] were needed to explain the numerical 
results. 

In the present paper, we have addressed this question, and we have obtained answers 
in completly unanticipated directions, as follows. We have performed 1RSB and 2RSB 
analyses for general Hopfield-type models storing ensembles of low-activity patterns [1],[2]. 
For an activity a = 1/2, these models are equivalent to Hopfield's standard model [5], and 
our 1RSB results should therefore merge with those of Crisanti et al. [3] as we take the 
limit a — > 1/2. It turns out that such 'confirmation en route' of the findings of Crisanti et 
al. could not be accomplished, and we are forced to conclude that their often quoted value 
q.irsb ~ 0.144 for the storage capacity of the standard model is in error — in particular 
since we have invited an independent check of our results, which was recently performed 
by Huyghebaert [12] using the bifurcation-finding software package 'AUTO'. 

Our main results for the a = 1/2 case are a* RSB ~ 0.138 186, and a^RSB _ q 13g lg7> 
These values are still lying well within the recently discovered reentrance phase of the 
Hopfield model [4]. That is, 1RSB and 2RSB approximations are found insufficient to 
bridge the gap between RS and simulation results. Moreover, we shall argue on the basis 
of the Parisi- Toulouse hypothesis [13] appropriatly adapted to the Hopfield model phase 
diagram that even the full hierarchical RSB scheme of Parisi [11] will not yield storage 
capacities as high as those reported from various simulation studies. Rather, as in the 
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SK-model, the reentrance phenomenon is expected to simply disappear in the full RSB 
solution. 

This renders the problem of how to explain discrepancies between theory and numerical 
experiment an open question again. 

For values other than a — 1/2 we obtain qualitatively similar results. Here, we have 
performed numerical evaluations only of the 1RSB theory, and we find the 1RSB storage 
capacity to lie again within the RS reentrance phase (as it should according to the Parisi- 
Toulouse hypothesis), aJ RSB ~ 0.495 at a = 0.1, well below the simulation result a & c im = 
0.585 ±0.01 [9]. 

Our paper is organized as follows. In Sec. 2, we introduce the generalized Hopfield 
model for the storage of low-activity patterns. In Sec. 3, we compute the replica free 
energy and evaluate it in the RS, and the 1RSB and 2RSB approximations, relegating 
details of the derivations to appendices. In Sec. 4 we present and discuss outcomes of a 
numerical analysis of the 1RSB and the 2RSB approximations for the Hopfield model, and 
of the 1RSB approximation for the generalized model at a — 0.1. In Sec. 5, we discuss 
analogies between the Hopfield model and the SK-model with ferromagnetic anisotropy 
[6] to put our results into a wider perspective. From an appropriate adaption of the 
Parisi- Toulouse hypothesis to the phase diagrams of the generalized Hopfield-type models, 
we conjecture that even the full hierarchical RSB solution of these models, presumably 
providing their exact solution, would yield results which are at variance with currently 
available simulation data. A concluding section is devoted to discuss the state of affairs 
that has thus emerged. 



In order to set the scene and to fix our notation, we shall here introduce our variant of 
the generalized Hopfield model for the storage of patterns of arbitrary activity. 

Let us denote by a the average fraction of active nodes in each pattern to be stored. 
We take a storage prescription of a generalized Hebbian form 



where 1 < i, j < N label the neurons of the net and 1 < [i < p enumerates the patterns. 
For the representation of the active (A) and inactive (/) neural states Sj, we take 



2 The Generalized Hopfield Model 



i ^ 



(i) 



N 




(2) 



given that the pattern statistics is 




A, with prob. a 
I, with prob. 1 — a 



(3) 
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The representation defined by (|2]) is known to be well adapted to the storage of patterns 
with statistics given in (|3]) [1],[2], using a generalized Hebbian storage prescription as in 
([I]), and supplementing it with a suitable threshold Moreover, it is known to saturate 
the well known Gardner bound [14] 

(4) 



2a\ lna| ' 

as a — > 0. The asynchronous dynamics of the model, defined by 

Si (t + At) = A 9 (hi(t) -#) + 1(1-0 (hi(t) - 0)) , (5) 

with hi(t) = J2j Jij s j(t) an d 0(x) the usual Heaviside function, is governed by the energy 
function 

H \ s \ = J V Si s i + ® ^2 Si - ^ 

i^j i 

Asynchronous Glauber dynamics, if adapted to (||D, converges to a Gibbs distribution over 
the space of neural states that is generated by (|). The next section is devoted to deal 
with the quenched randomness in the couplings due to the stored patterns. 



3 The Replica Free Energy 
3.1 General Theory 

As usual, to investigate the thermodynamics of systems with quenched randomness, one 
has to compute the quenched free energy —/3f(/3) = (InZ)^, where Z is the partition 
function at fixed disorder and where (. . .)^ denotes an average over disorder according to 
its distribution. The computation utilizes the replica identity 



<lnZ) € = hm-ln(Z% 

n—*0 fi 



(7) 



In the details of the calculation we follow Amit et al. [7]. Assuming that the system 
state has macroscopic correlations only with a finite number of patterns, say £ u , v = 
1, . . . , / , we obtain the replica free energy through standard arguments [7], by averaging 
over the remaining patterns £ M : 



nf 



1 



E«) 2 + 7^ ^ det (1 - /3q) + Pa £ r ab q ab - \ (In Z) c 



2(3 



a<b 



(8) 



Here Z is a replica partition function 



Z=£exp(/?($>" 

{s a } I V a 



a<b Z a ) 



(9) 
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corresponding to a replicated single-site Hamiltonian of the form 



H 



£* a 



rv 

a<b 1 a 



We have introduced overlaps with the condensed patterns 

1 



a N 

and the matrix q of Edwards- Anderson order parameters, with elements 



q a b 



-E 

AT Z— i 



N 



s a s b 



(10) 



(11) 



(12) 



with 1 < a, b < n labeling the replicas. In mean-field theory, the order parameters must 
satisfy the fixed point equations 



q a b 



<r<s a »e, f = i,...,Z, 



1 n 



((s a s b ))^, l<a,b<n, 



(13) 



in which (. . .) without subscript denotes a Gibbs average performed with the Hamiltonian 
H in (pT0| ) while (. . designates an average over the condensed patterns The matrix 
r with elements is simply related to the q-matrix. One has 



(3r aa = -(1- (3r ab 



[l-^)- b \ a<b, 



(14) 



with 1 the nxn unit matrix. It is understood that an analytic continuation to non-integer 
n and the n —>■ limit are eventually to be taken. 

Equations fll3"D-(|T4"D are usually solved by making an ansatz concerning the transfor- 
mation properties of the saddle-point values of the order parameters m v a and q ab under 
permutation of replicas. 



3.2 The replica symmetric approximation 

The first and most natural ansatz is of course that exhibiting complete replica symmetry 
(RS): 

m v a = m h ', q aa = q, q ab = q, a ^ b, (15) 

the replica symmetry of q being inherited by r due to (0). This ansatz allows for an 
easy evaluation of all terms appearing in (H), ([13| ) and flliD , as well as for an analytic 
continuation of the results to n — > 0. As the n —>■ limit is taken one gets 

/ = ^EK0 2 -^<m^ 

+ | (- In (1 - m - q)) - l _ [3 q r _ ) + 2/?rg - 0rq\ , (16) 
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where m v , q and q satisfy 

m v = (C(s)) z ,e 

(17) 

Here (. . .) without subscript denotes a 'thermal average' performed over the Gibbs distri- 
bution generated by the single-site Hamiltonian 



H 



Y,m u - & + y/cir z 



s 2 -(/3(2f-r)-l) 



while Z is the corresponding partition function, and (. . .) z ^ a combined average over a 
zero mean unit variance Gaussian z and the £ v according to their distribution. Moreover, 
we have 

r= H-W- g )P W-'^T^M^- (19) 

The computation leading to these equations are standard, and we shall not document 
them here. 

The RS solution fails to be thermodynamically stable as the temperature is lowered 
through the AT-line [15], given by 

T 2 = j^y 2 (((s 2 )-(s) 2 f) z ,s, (20) 

where we have introduced the 'response parameter' C = j3 (q — q). The RS Hopfield model 
results are recovered by taking •& = and A = —I = 1 in (JTB|) - (P0|) • 

To improve upon the results of the RS approximation in the region where RS is known 
to be broken according to the AT criterion [15], one can follow Parisi's scheme of hierar- 
chical replica symmetry breaking [10,11]. In what follows, we present the first two steps 
of Parisi's approximations for the generalized Hopfield model. Details of the calculation 
are to be found in appendices A and B for the 1RSB and the 2RSB approximations 
respectively. 

3.3 The 1RSB approximation 

In the 1RSB ansatz, one assumes that the overlaps m v a still exhibit the full invariance 
with respect to permutations of replicas, 

m u a = m u , a = l,...,n, (21) 

whereas the Edwards-Anderson matrix q acquires the following structure, 



Qab 



q, a = b, 

q h \a-b\<m, (22) 
go, otherwise. 
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Here m < n is a partitioning parameter that is to be determined from a stationarity 
condition for the free energy. Formally, we may express q in terms of a tensor product 
structure 



q 



- qi) l n + (gi - g ) 1^ ® e m e r 



(23) 



where denotes a fc-dimensional unit matrix and e£ = (1, 1, . . . , 1) a transposed column 
vector with k elements identical 1, that is, e^e^F is nothing but a k x k matrix completely 
filled with one's. The matrix 1 — /3q clearly has the same type of tensorial structure, 
and so does r, because it is simply related with its inverse. These observations allow a 
fairly straightforward evaluation of all terms appearing in ©-(0), as well as of a 1RSB 
formulation of the saddle point equations fll3|) . We get (for details, see appendix A) 



f(m u ,q ,q 1 ,q,r ,r 1 ,r; m) 



m 



v\2 



1 

[3m 



ln ( Z r 



a . 
+ 2 1 



Q qo (3m 
for the free energy, and 



In 



Q 



9i 



- In Q 91 + (32fq + (3r iqi (m - 1) - /?r g m j (24) 



qo 



Qi 




(25) 



for the fixed point equations. There is an extra equation due to the stationary condition 
on / with respect to the partitioning parameter m, which can be expressed as follows: 



1 

(3m 



ln (Z 



Z m ln Z r 



(3m 



Z m ) 



a 

+ 2 \Q 



Qo 



</0 



H—LJQ 

Q qi (3m \Q 



In (f2^ - p6[) , we have introduced the auxiliary quantities Q qi defined as 



Q 



m-Qi) 



(26) 



(27) 



8 



H. SteSan and R. Kiihn: 



The elements of the r-matrix, are given by 

r = Ar 

n - r = Ari 
P(2f - n) = (3Af 



Ago 

Agi 

QgoQqi 
1 



Qqi 



(28) 



where Ag = go an d Ag x = gi — go- The quantity Z, finally, denotes the partition function 
corresponding to the single-site Hamiltonian 



H = - 



Y, m "C + \/ttAroz + \JaAn z x - ■& 



a 



(29) 



The notation used for the averaging brackets conforms to that introduced in the previous 
subsection. 

The numerical solution of these equations will be discussed in Sect. 4 below. The case 
of the standard Hopfield model is recovered by taking A = — I = 1 and ■& — 0. 



3.4 The 2RSB approximation 

The 2RSB approximation is obtained from the 1RSB scheme by endowing the m x m 
diagonal submatrices of q with a structure akin to that arrived at when breaking RS for 
the first time in the full matrix. Formally, 

Q = (q ~ Q2) In + (g2 - gi) 1 jl <8> e m2 e^ 2 + (gi - g ) 1 j_ ® e mi e^ 1 + g e n ej. (30) 

The tensorial structure of q is inherited by 1 — j3 q and by r for the same reasons as in 
the 1RSB case. Moreover, as in the 1RSB approximation, one keeps replica symmetry for 
the overlaps m v a . This leads to the free energy 



f(m u ,q ,q 1 ,q 2 ,q,r ,r 1 ,r 2 ,f; m 1 ,m 2 ) = - J2(m u ) 2 - (in ^ (z m2 )™ 2 ^ 



z,e 



2^ Q qo (3m, \Qj (3m 2 \Q q J (3 



+ -[ - — h. I ^ \ - — In ( -rr- ) + -lnQ 92 

+ (32fq + f3r 2 q 2 (m 2 - 1) + finqx (m x - m 2 ) - [3r q Q m^j (31) 
and the saddle point equations 



^ = ( r - — r- — v ^/ 2 ; " ) (32) 
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go = 



Qi 



12 



c=m- 12) 



T± (Z m 2(s)) 
22 (Z m2 )_ 



Zl 



2'm 2 \ m 2 

22 



Zl 



Z2 



(Z m 2) 
\ / 20 



2j m 2 



Z2 



Zl 



^m2 \ m 2 x ' 2 



^TO2 \ m -> 

22 



z,e 



21 



-J_/ 

v / aAr^ \ 



£1712 \ m 2 

22 



^"12 \ m 2 

22 



21 



2,£" 



(33) 



(34) 



(35) 



(36) 



The partitioning parameters mi and m2 are determined by stationarity conditions on /, 
leading to two further fixed point equations, namely: 



f3m\ 



In 



\ m 2 
22 



2,^ 



1 



/5m 2 \ 



Z m2 ) m ' 2 In (Z m2 
22 \ ' -.2 

2'm 2 \^2 
' 22 



21 



2,e 



In 



Q 



9(> 



2 Q91 P m i 



(37) 



91 . 



mi , / , _ . Ull ( Z m 2 In Z m 2 \ 



21 



/5m 2 \ ^ (2m2\^ \ I P m 2 \ / lz m2 Y 2 

' Z2 I Zl 2 , e \ ^ /z 2 / 2l ,, 



+ 2^1 Q92 UJJ ( j 



The r elements, finally, are algebraically related to the elements of q: 

Ago 



r = Ar c 



^90 
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An = 



ri - r 



Qq {) Qi 

Ag 2 



r 2 - ri 



Ar 2 = 



P(2f - r 2 ) 



/3Ar = 



1 



(39) 



<2, 



Here Ag = qo, and Ag^ = qi — q%-\ for i = 1,2, while 

Q q2 = 1 ~ P(q ~ q2) , Q qi =Q q2 - (3m 2 (q 2 -q l ) , Q qo = Q qi - /3mi(gi - g ) ■ (40) 

As in the previous subsection, Z denotes a partition function corresponding to a single-site 
Hamiltonian, namely 



Conventions regarding averaging brackets are the same as before. 

From the structure of the 1RSB and the 2RSB equations, a formulation of the infinite 
RSB scheme is fairly easily obtained. Since we have not evaluated this limit numerically, 
we will not reproduce the corresponding equations here. The interested reader will find 
the details in [16]. 

4 Results 

We have solved the fixed point equations corresponding to the RS, the 1RSB and the 2RSB 
approximations for the Hopfield model with a = 1/2, as well as those corresponding to 
the RS and the 1RSB approximations for the generalized model at a = 0.1. In both 
cases, simulation data are available for comparison with theoretical results [7-9] . The RS 
approximations reproduce previously known results as they should. 

As usual, the numerics simplifies considerably in the T = limit, because the inner- 
most Gaussian averages in the saddle point equations can be performed analytically in 
this limit, giving simple expressions in terms of error functions. Moreover, it can be shown 
that, as this limit is taken, the partitioning parameters m and m\, m 2 of the 1RSB and the 
2RSB approximations enter the theory only through the scaled combinations D = f3m 
and Di = [3 mi, D 2 = (3 m 2 , which remain finite as the j3 — > oo limit is taken. In the case 
of the 1RSB approximations, this has already been noticed by Crisanti et al. [3]. 

4.1 The standard model at a = 1/2 

In the case of the standard model, we have A = —I = 1 and ■& — 0. The full phase diagram 
in the RS approximation is well known [7]. In Fig. 1 we present, for later reference, an 
enlarged portion of it, exhibiting the boundary of the retrieval phase at low temperatures, 
as well as its AT-line. A noteable feature here is the reentrant behaviour signified by a 
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backbending of the transition line for T < T(a max ) = 0.024, previously discovered by 
Naef and Canning [4]. Moreover, the AT-line below which the RS approximation fails 
to be thermodynamically acceptable is seen to meet the RS phase boundary just slightly 
above T(a max ), where reentrant behaviour begins. We will return to discussion of these 
features in Sec. 5 below. 

Fig. 1 

Knowing that replica symmetry must be broken below the AT-line, we have analyzed 
the 1RSB approximation of the model. Fig. 2 shows the order parameters m 1 , gi, and 
go (q = 1) at T — 0.02, for a range of a values that cross the AT-line at «at — 0.1376. 
As is to be expected q\ and go become different, as a is increased through «at- The 
figure exhibits both stable and unstable solutions of the order parameters. The stable 
and unstable branches of these order parameters all coalesce at a ( i RSB (T) ~ 0.13819, 
signifying the tangent-bifurcation that marks the boundary of the retrieval phase at this 
temperature in the 1RSB approximation. At zero temperature, T = 0, we find the 
following critical parameters 

a c ~ 0.138186 489 5 
m 1 ~ 0.966 77 
g ~ 0.996 48 
C ~ 0.052 89 

D ~ 36.78 (42) 

These values deviate considerably from those previously reported by Crisanti et al. [3] 

a c ~ 0.144 

m 1 ~ 0.982 

C ~ 0.111 

D ~ 0.03 (43) 



which they obtained through a Monte-Carlo minimization of the free energy function (|2 
rather than by deriving and solving the associated fixed point approximations. 

Fig. 2 

In view of this discrepancy, we have performed several internal consistency checks 
of our results. Since our expression for the free energy is the same as in [3], a possi- 
ble discrepancy could arise due to errorneous expressions for the fixed point equations. 
However, our 1RSB fixed point system can in principle also be derived from the 2RSB 
approximation by either taking the limit rri2 1 or the limit m\ — > n. Both checks 
confirmed the expressions given in (|25|-|29D. Moreover, we have some internal consistency 



checks by isolating asymptotics of various integrations analytically, with essentially no 
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change on the results. We have also omitted the stationarity requirement with respect 
to the partitioning parameter m, treating D = (3 m as a free parameter, and computing 
a c at T = as a function of D. The result is shown in Fig. 3. The capacity a c (D) 
never increases beyond ct^ {SB (D) max ~ 0.1382, approaching — as it should — the replica 
symmetric capacity o RS = 0.137905 as D — > or D — > oo, albeit in the latter case slowly. 
Even aJ RSB (-D) max is found to be slightly smaller than o RS (T) max ~ 0.138 188 5. Lastly, 
an independent check of our results was recently obtained by Huyghebaert [12], using the 
bifurcation finding software package 'AUTO', and confirming our results to an accuracy 
of 9 significant digits. 

Fig. 3 





a c 


m 1 


D 




D 2 


f = u 


s 


RS 


0.137905 566 


0.967417 








-0.501445 395 


-0.001445 


1RSB 


0.138186489 


0.966 777 


36.783 






-0.501446 051 


-0.000104 


2RSB 


0.138187733 


0.966 776 




2.406 


38.320 


-0.501446125 


-0.000 097 



Table 1: Retrieval boundary at T = in RS, 1RSB and 2RSB approximations 
for the standard model with a = 1/2. Also given are values for (free) energy u and 
entropy s. 

We have also considered the 2RSB approximation in the T — > limit. Because of 
additional integrations that need to be performed numerically, these results are inherently 
less precise than those for the 1RSB scheme. They are collected in Tab. 1. 

Note that the 2RSB approximation gives only a rather slight increase in the T = 
storage capacity, which is still within the RS reentrant phase, i.e., a^ RSB < o RS (T) max . 
Fig. 4 shows the Parisi function, as a function of the rescaled partitioning parameters 
Di = (3rrii, in the 1RSB and the 2RSB approximations, respectively. 

Fig. 4 



4.2 The general model at a = 0.1 

For the general model (a different from 1/2), there is no longer a symmetry between the 
active and inactive neural states. In what follows we shall exclusively deal with the a — 0.1 
case, for which simulations results are available for comparision [9] . A non-zero threshold 
has to be introduced, and it must be optimized in order to yield the largest retrieval 
region. In Fig. 5, we show the RS boundary of the retrieval region for the generalized 
model at a — 0.1, for various values of the threshold The optimal threshold, yielding 
the largest T = capacity is found to be ^ opt ~ 1.825 57 for this system. 
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m 1 


C 


D 


f = u 


s 


RS 


0.484151834 


0.836 979 


0.176 688 




-0.508 557635 


-0.004 886 


1RSB 


0.495 030 302 


0.828196 


0.055155 


3.523 


-0.509 302481 


-0.000 406 



Table 2: Retrieval boundary at T = in the RS and the 1RSB approximations 
for the generalized model at a = 0.1 and with optimal threshold i? = 1.82557. 

In principle, the envelope of the retrieval regions attainable with continuously varying d 
gives the ultimate boundary for retrieval. This would require d to vary with temperature, 
$ = $(T). Note the strong reentrant behaviour in case of near optimal threshold, clearly 
visible in the present case without amplification. 

As in the standard model, the RS solution fails to be acceptable for T < Tat (a), 
given by (p0f) , and depicted in Fig. 6 for $ = $ opt . Again, exactly as in the standard 
model, the AT-line is seen to meet the RS phase boundary just slightly above T(a max ) 
below which the RS phase boundary bends back to lower values of a. For the present 
case T(a max ) ~ 0.24 at a max — 0.495 17, whereas the T = capacity is 

af s ~ 0.48415 (44) 

We have investigated the 1RSB approximation of this model at T = 0, with results 
collected in Tab. 2. 

Fig. 5,6 

The q values for this case are at q = q 1 ~ 0.929 310 and q ~ 0.891226 compared to 
the replica symmetric q = q ~ 0.924 081. While the relative increase in computed T = 
storage capacity due to 1RSB corrections is significantly larger than in the standard 
model (roughly 2% as opposed to only 0.15% in the standard model), it does still give 
a capacity within the reentrant phase, and far below the number a s c im = 0.585 ± 0.01 
reported from numerical simulations [9]. Again we have checked our results by treating 
D as an independent parameter, not fixed by a stationarity requirement, with results 
qualitatively similar to the case of the standard model; see Fig. 7. 

Fig. 7 

Though we have not analyzed the 2RSB approximation, we expect the outcome of such 
an analysis to be qualitatively similar to the standard case: there will be an additional 
slight increase in the T = storage capacity, but it will still be smaller than a^ s (T) max , 
i.e. still be within the reentrant phase. 

A final note here concerns the sharp bends in the RS phase diagrams, at temperatures 
above T(a max ). They are due to the fact that the RS retrieval phase loses stability in 
different directions giving way to different frozen phases, depending on whether a < a* 
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or a > a*, where a* denotes the loading level at which the sharp bend occurs. Since the 
non- retrieval phases are not so much of concern to us in the present context, we will not 
elaborate on this point, however. The interested reader may consult [1] and [16] on this 
matter. 



In the previous section, we have seen that 1RSB or 2RSB approximations to the mean-field 
solution of generalized Hopfield-type models yield a slight increase of the storage capacities 
of these models, but this increase was found to be much smaller than probably expected 
from simulation results [7-9], or previously reported [3]. In all cases , the resulting storage 
capacity was found to be smaller than that maximally attainable in the RS approximation 
at finite temperature. In Sect. 4, we have also evaluated T = energies and entropies; 
cf. Tab. 1 and 2. The results show that the 1RSB and 2RSB approximations are still not 
thermodynamically acceptable at very low temperatures near the respective a c , because 
the T = entropies turn out to be negative, which is strictly forbidden in a system with 
discrete variables. 

Internal energy u and entropy s are computed from the relations u = ^M- and s = 
flu — p f in units of A; B . The computations are straightforward, if perhaps tedious in 
details. For the T = entropy, one obtains 



which is formally independent of the degree of approximation in the finite-step RSB 
scheme, provided the response parameter C is defined as C = /3 (q — q(l)), with q the 
diagonal entry in the matrix of Edwards-Anderson order parameters, and q(l) the off- 
diagonal entry in the innermost blocks, that is q(l) = q in the RS approximation, and 
q(l) = qt in the kRSB scheme, k — 1, 2, . . .. Thus as long as C(T = 0) is non-zero, the 
T = entropy will come out negative. 

The same formal independence for the zero entropy expression of the degree of ap- 
proximation in the Parisi scheme is observed for the SK-model, where 



with C = (3(q- q(l)) = P (1 - q(l)), and q(l) = q k in the kRSB scheme. 

As a consequence, it is to be expected that no finite approximation in Parisi's ap- 
proximating scheme will yield thermodynamically acceptable solutions at T = for the 
retrieval phases of generalized Hopfield models, just as in the case of the SK-model, where 
only the full hierarchical scheme of infintely many levels of RSB gives a Parisi function 
q(x) that is sufficiently smooth on the D = P x scale to produce a vanishing C in the 
T = limit, and thereby a vanishing zero temperature entropy. 

Further evidence for the analogy between the retrieval phases of generalized Hopfield 
models and the magnetic phase of the SK-model with ferromagnetic anisotropy (J^) = 



5 Relation with RSB in the SK-model 




(45) 
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J /N comes from comparing lines of constant magnetization in a RS approximation; see 
Fig. 8. 

Fig. 8 

As in the Hopfield model, reentrance is observed in the RS approximation of the SK- 
model phase diagram. Moreover, in both models, the AT-line is seen to intersect the 
constant-magnetization lines slightly above the temperature where they begin to bend 
back (to larger Jo in the SK-model, to smaller a in the Hopfield model). Thus, reentrant 
behaviour as observed in RS approximations is found to be AT unstable in both models, 
in a strikingly similar fashion.[| 

Now according to the Parisi- Toulouse hypothesis [13], one effect of RSB in the SK- 
model is, roughly, to freeze the value of the magnetization as a function of temperature. 
That is, in the full RSB solution of the SK-model, the iso-magnetization lines in the 
phasediagram 7 will be verticals below the AT line. This statement is believed to be 
exact for the m = line, i.e., the phase boundary, and to constitute a very precise 
approximation otherwise [13]. 

By analogy, and in view of the great similarity of the analytic structure, the same 
is expected to hold in the case of generalized Hopfield models. As a consequence, the 
retrieval phase boundary — as the envelope of iso-overlap lines with non-zero m v - 
should in the full RSB scheme turn out to be vertical (or very close to a vertical) below 
the point where the AT line touches the RS phase boundary. 

This hypothesis is completely in accord with the results of our 1RSB and 2RSB analy- 
ses, which showed that the T = transition point a c is shifted to slightly higher values, 
closer to the point where they are expected to be if the hypothesis were true, and never 
beyond the abscissa a RS (T) max of the reentrant point, in contrast to previously reported 
results [3]. 

6 Summary and Discussion 

We have studied effects of RSB in generalized Hopfield-type models of attractor neural 
networks. We have obtained 1RSB and 2RSB corrections to RS results, which are much 
smaller than expected from simulation results [7-9] or previously reported for the standard 
model at a = 1/2 [3]. In all cases the 1RSB and 2RSB storage capacities obtained were 
found to be smaller than those attainable in the RS approximation at finite temperature. 
Our results were found to be consistent with what is to be expected if the Parisi-Toulouse 
hypothesis about the nature of the Parisi function q(x) [11,13] would hold for Hopfield- 
type models in the same manner as it does in the SK spin glass model. On the basis of 
this hypothesis, we conjecture that the reentrance phenomenon observed in RS analyses 
of Hopfield-type models would simply disappear in a full hierarchical Parisi RSB solution 

1 There is a mapping of the replica symmetric Hopfield model onto the replica symmetric SK-model; 
for details see [16]. 
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of these models. 

This state of affairs raises the question of how to reconcile discrepancies between theory 
and numerical experiments in these systems. Two possible explanations come to mind, 
and both are, we think, worth checking. 

One possibility is that T = O-Monte Carlo Dynamics gets trapped in energy valleys 
which are surrounded by nonextensive energy barriers, AE ~ N u with < v < 1. 
The existence of such nonextensive energy barriers between thermodynamically unstable 
retrieval states and the spin glass phase might well invalidate conventional (exponential) 
finite-size-scaling expressions for first order phase transitions, on which the analyses of 
simulation data [7,9] were based. Here we should, however, remark that Miiller [17] - 
knowing of our results — has recently performed Monte Carlo simulations which would 
confirm our values for a c on the basis of a conventional finite-size-scaling analysis of his 
data. 

A second possibility concerns the existence of dynamically frozen phases not detectable 
in equilibrium treatments. Discrepancies analogous to those between our results and those 
of simulations have, indeed, recently been observed in the case of the binary perceptron 
[18], the p-spin interaction spin glass [19], and in the case of fluctuating manifolds in ran- 
dom media [20], where dynamically frozen phases were observed in regions of parameter 
space in which static approaches yielded ergodic phases. 
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A The 1RSB Approximation 

In this appendix, we present the main ideas that go into the evaluation of the replica 
free energy (§) in the 1RSB approximation, and into the derivation of the corresponding 
1RSB version of the fixed point equations (|13). The 1RSB approximation is based on 



the ansatz (p3|) for the matrix of Edwards-Anderson order parameters, in which m is a 
partitioning parameter for the set of n replica that is to be determined from a stationarity 
condition on /. Clearly, the structure of q is inherited by 1 — Pq, 



1 - /3q = (1 - p(q - qi)) l n - p(qi - q ) li <g> e m e m - pq e n e n , (47) 

as well as by r in virtue of (|Tj[). Using this structure, we have to evaluate the various 
terms appearing in the free energy (^|). 

(i) In order to evaluate lndet(l — Pq), we diagonalize d47|) . This is accomplished 



by noting that l n = In (g) l m , and e n ej = enelL <g> e m e^, and that matrices with 
tensor product structure can be diagonalized separately in each tensor product component. 
Matrices of the form e^e^ have one eigenvalue k and a (k — l)-fold eigenvalue zero. Since 
they trivially commute with the corresponding unit matrices 1^, the full spectrum of (|47[) 
is readily obtained to yield 

det(l-/3q) = (1 - p(q - qi) - Pm(q 1 - q ) - npq ) 
(1 - P(q - q x ) - pm(q 1 - g )) m 

(l-P(q- qi )f-%. (48) 

(ii) Next, the term Y. a <b r ab ( lab appearing in (|) is evaluated by endowing r with the 
same block structure as q, and by parametrizing it analogously. This gives 

= 2 [2H + (m - l)nqi + (n - m)r q ] (49) 

(iii) In view of (]14[) , to express the elements of r in terms of those of q, we have to 
invert 1 — 3q. This is done by noting that the matrices and e^e^ form a closed algebra, 
since e^e^e^e^ = /ce^e^. Thus the inverse of 1 — Pq must be of the same structure as 
1 — Pq itself, albeit with different coefficients in front of the three tensor product matrices 
appearing in fliT]) . These coefficients are computed from the condition that the product 
of 1 — Pq and its inverse should give a unit matrix. This yields (PSf), with the Q qi defined 
by p. 

(iv) Finally, to evaluate the single-site replica partition function corresponding to the 
Hamiltonian (|10D , we have to decouple the replicated spins coupled through the term 
J2 a <b r abS a s b . With r of 1RSB form analogous to (|3|), we get 




T.rabS a s h = \ 

a<b Z 



n n/m / m \ 2 / n \ 

(2r - ri ) ]>> a ) 2 + (n -ro)E E s (^ + a + ro s a 

a=l k=l \a=l / \a=l / 



(50) 
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Gaussian linearization of the last term in (pUD, using a Gaussian variable z, will produce 
n/m identical uncoupled blocks of size m in the evaluation of @. Within these blocks 
there is still a coupling between spins due to the next-to-last term in (|50|). These are 
decoupled using a Gaussian z\. This then yields 



(lnZ) e =n(\n(Z m )?l > (51) 
\ / 

in (|8|), as n — > 0, with Z the partition function corresponding to the on-site Hamiltonian 
(p9l). Similar linearization techniques are readily seen to produce the 1RSB fixed point 



equations (|25| ) from (|1^), as the n — > 0-limit is taken. 

Collecting all items so far computed, we obtain the 1RSB expression (|24]) . Stationarity 
of this expression with respect to the partitioning parameter m requires ( PS| ) to hold, which 
completes our derivation of the 1RSB approximation. 



B The 2RSB Approximation 

In the present appendix, we sketch the evaluation of (§) in the 2RSB approximation, and 
the derivation of the corresponding 2RSB version of the fixed point equations (0). The 
following outline completely parallels that of Appendix A, and we will not repeat the 
arguments in detail. 

The 2RSB approximation is based on the ansatz ( p0|) for the matrix of Edwards- 
Anderson order parameters, in which mi and m 2 are partitioning parameters which are 
to be determined from a stationarity condition on /. Again, the structure of q is inherited 
by 1 - /3q, 

1 - (3q = (1 - (3{q - q 2 )) 1„ - (3{q 2 - qi) 1 j_ ® e ma e^ 2 

-P(qi - go) 1-2- ® ~ PVo e„e^. (52) 



as well as by r in virtue of (|l^ 

(i) In order to evaluate lndet(l — /3q), we diagonalize (^2j) along the lines outlined in 
Appendix A. This yields 

det(l-^q) = {I - (3{q - q 2 ) - I3m 2 (q 2 - q{) - (3mi{qi - q ) - (3nq ) 
(1 - (5(q-q 2 ) - (3m 2 (q 2 - q t ) - Pm x {qi -g ))) mi 

n n_ 

(1 - f3{q - q 2 ) - (3m 2 (q 2 - q^))^ *n 

{l-P{q-q x )) n -^. (53) 

(ii) The term Y«x<b r ab Qab appearing in (|8|) is evaluated by endowing r with the same 
block structure as q, and by parametrizing it analogously. This gives 

r ab q a b = ~[ 2rq + {m 2 - l)r 2 q 2 + (ra x - m 2 )r 1 qx + (n- mi)r g ] (54) 

a<b 1 
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(iii) To express the elements of r in terms of those of q, we have to invert 1 — /3q. 
This is done as outlined in Appendix A, and yields (|39|), with the Q qi defined by (^0|). 

(iv) Next, to evaluate the single-site replica partition function corresponding to the 
Hamiltonian (|10l), we have to decouple the replicated spins coupled through the term 



Ha<b r abS a s . With r of 2RSB form analogous to 



we get 



a<b 



n/m,2 



(2f - r 2 ) $> a ) 2 + (r 2 - n) £ (£* 

a=l k=l a=l 



1112 



(k—l)rri2+a 



n/mi mi _ n 

Hn-ro) E (E^" 1)mi+a ) + n>(E* a 



k=l a=l 



a=l 



(55) 



This structure suggests an iterative Gaussian linearization scheme as in the case of the 
1RSB approximation dicussed in the previous appendix. A Gaussian z is introduced to 
decouple spins in different blocks of size mi, and creates n/mi identical independent, i.e., 
uncoupled blocks of this size. Within a block of size mi, there are mi/m.2 identical blocks 
of size m 2 , which are decoupled through a Gaussian Z\. Finally the m 2 spins within each 
of these smaller blocks are decoupled using a Gaussian z 2 . This results in 



n 



ln((Z m2 ),7 



1 



(56) 



in (Rl), as n — >• 0. Here Z is the partition function corresponding to the on-site Hamiltonian 



fl4ip. Again, similar linearization techniques are seen to produce the 2RSB fixed point 
equations from flTBD, as the n — > 0-limit is taken. 



Collecting all items, we obtain the 2RSB expression fl3T|). Stationarity of this expres- 
sion with respect to the partitioning parameters mi and m 2 requires (|37]) and (^) to 
hold, which completes our derivation of the 2RSB approximation. 



